Pandas Profiling Report

Dataset statistics

Number of variables	12
Number of observations	2891
Missing cells	48
Missing cells (%)	0.1%
Duplicate rows	0
Duplicate rows (%)	0.0%
Total size in memory	271.2 KiB
Average record size in memory	96.0 B

Variable types

NUM	8
CAT	4

Reproduction

Analysis started	2020-06-11 17:50:24.854860
Analysis finished	2020-06-11 17:50:37.899862
Duration	13.05 seconds
Version	pandas-profiling v2.8.0
Command line	`pandas_profiling --config_file config.yaml [YOUR_FILE.csv]`
Download configuration	config.yaml

Warnings

`City` has a high cardinality: 567 distinct values	High cardinality
`Female Population` is highly correlated with `Male Population` and 2 other fields	High correlation
`Male Population` is highly correlated with `Female Population` and 2 other fields	High correlation
`Total Population` is highly correlated with `Male Population` and 2 other fields	High correlation
`Foreign-born` is highly correlated with `Male Population` and 2 other fields	High correlation
`State Code` is highly correlated with `State`	High correlation
`State` is highly correlated with `State Code`	High correlation
`City` is uniformly distributed	Uniform

City
Categorical

HIGH CARDINALITY
UNIFORM

Distinct count	567
Unique (%)	19.6%
Missing	0
Missing (%)	0.0%
Memory size	22.6 KiB

Bloomington	15
Columbia	15
Springfield	15
Concord	10
Columbus	10
Other values (562)	2826

Common Values
Length

Value	Count	Frequency (%)
Bloomington	15	0.5%
Columbia	15	0.5%
Springfield	15	0.5%
Concord	10	0.3%
Columbus	10	0.3%
Albany	10	0.3%
Norwalk	10	0.3%
Westminster	10	0.3%
Portland	10	0.3%
Aurora	10	0.3%
Other values (557)	2776	96.0%

Length

Max length	47
Median length	9
Mean length	9.10307852
Min length	2

State
Categorical

HIGH CORRELATION

Distinct count	49
Unique (%)	1.7%
Missing	0
Missing (%)	0.0%
Memory size	22.6 KiB

California	676
Texas	273
Florida	222
Illinois	91
Washington	85
Other values (44)	1544

Common Values
Length

Value	Count	Frequency (%)
California	676	23.4%
Texas	273	9.4%
Florida	222	7.7%
Illinois	91	3.1%
Washington	85	2.9%
Colorado	80	2.8%
Arizona	80	2.8%
Michigan	79	2.7%
North Carolina	70	2.4%
Virginia	70	2.4%
Other values (39)	1165	40.3%

Length

Max length	20
Median length	8
Mean length	8.426841923
Min length	4

Median Age
Real number (ℝ_≥0)

Distinct count	180
Unique (%)	6.2%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	35.49488066413006
Minimum	22.9
Maximum	70.5
Zeros	0
Zeros (%)	0.0%
Memory size	22.6 KiB

Quantile statistics

Minimum	22.9
5-th percentile	28.8
Q1	32.8
median	35.3
Q3	38
95-th percentile	42.6
Maximum	70.5
Range	47.6
Interquartile range (IQR)	5.2

Descriptive statistics

Standard deviation	4.40161673
Coefficient of variation (CV)	0.1240070863
Kurtosis	4.164544019
Mean	35.49488066
Median Absolute Deviation (MAD)	2.6
Skewness	0.6466184403
Sum	102615.7
Variance	19.37422984

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
35.7	50	1.7%
33.4	48	1.7%
36.8	45	1.6%
34.1	45	1.6%
33.1	45	1.6%
34.5	44	1.5%
38.1	42	1.5%
35	40	1.4%
35.3	40	1.4%
35.5	40	1.4%
Other values (170)	2452	84.8%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
22.9	5	0.2%
23	4	0.1%
23.5	5	0.2%
23.6	5	0.2%
23.9	5	0.2%

Value	Count	Frequency (%)
70.5	3	0.1%
48.8	5	0.2%
47.9	4	0.1%
47.6	4	0.1%
47.4	5	0.2%

Male Population
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct count	593
Unique (%)	20.5%
Missing	3
Missing (%)	0.1%
Infinite	0
Infinite (%)	0.0%

Mean	97328.42624653739
Minimum	29281.0
Maximum	4081698.0
Zeros	0
Zeros (%)	0.0%
Memory size	22.6 KiB

Quantile statistics

Minimum	29281
5-th percentile	32290
Q1	39289
median	52341
Q3	86641.75
95-th percentile	296902.6
Maximum	4081698
Range	4052417
Interquartile range (IQR)	47352.75

Descriptive statistics

Standard deviation	216299.9369
Coefficient of variation (CV)	2.222371667
Kurtosis	209.8137883
Mean	97328.42625
Median Absolute Deviation (MAD)	15991
Skewness	12.73559694
Sum	281084495
Variance	4.678566272e+10

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
40601	10	0.3%
33993	10	0.3%
34311	5	0.2%
264893	5	0.2%
40282	5	0.2%
65113	5	0.2%
38827	5	0.2%
48984	5	0.2%
1149686	5	0.2%
55639	5	0.2%
Other values (583)	2828	97.8%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
29281	5	0.2%
29995	5	0.2%
30007	5	0.2%
30193	4	0.1%
30758	5	0.2%

Value	Count	Frequency (%)
4081698	5	0.2%
1958998	5	0.2%
1320015	5	0.2%
1149686	5	0.2%
786833	5	0.2%

Female Population
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct count	594
Unique (%)	20.6%
Missing	3
Missing (%)	0.1%
Infinite	0
Infinite (%)	0.0%

Mean	101769.63088642659
Minimum	27348.0
Maximum	4468707.0
Zeros	0
Zeros (%)	0.0%
Memory size	22.6 KiB

Quantile statistics

Minimum	27348
5-th percentile	34163
Q1	41227
median	53809
Q3	89604
95-th percentile	315853.35
Maximum	4468707
Range	4441359
Interquartile range (IQR)	48377

Descriptive statistics

Standard deviation	231564.5726
Coefficient of variation (CV)	2.2753799
Kurtosis	227.6330518
Mean	101769.6309
Median Absolute Deviation (MAD)	15771
Skewness	13.32044486
Sum	293910694
Variance	5.362215127e+10

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
35801	10	0.3%
97350	5	0.2%
88232	5	0.2%
508602	5	0.2%
60664	5	0.2%
56151	5	0.2%
36182	5	0.2%
118443	5	0.2%
44199	5	0.2%
96944	5	0.2%
Other values (584)	2833	98.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
27348	5	0.2%
31238	4	0.1%
31456	4	0.1%
32173	3	0.1%
32397	5	0.2%

Value	Count	Frequency (%)
4468707	5	0.2%
2012898	5	0.2%
1400541	5	0.2%
1148942	5	0.2%
826172	5	0.2%

Total Population
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct count	594
Unique (%)	20.5%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	198966.77931511588
Minimum	63215
Maximum	8550405
Zeros	0
Zeros (%)	0.0%
Memory size	22.6 KiB

Quantile statistics

Minimum	63215
5-th percentile	67271
Q1	80429
median	106782
Q3	175232
95-th percentile	618619
Maximum	8550405
Range	8487190
Interquartile range (IQR)	94803

Descriptive statistics

Standard deviation	447555.9296
Coefficient of variation (CV)	2.249400283
Kurtosis	219.2158805
Mean	198966.7793
Median Absolute Deviation (MAD)	32640
Skewness	13.04462299
Sum	575212959
Variance	2.003063102e+11

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
68097	10	0.3%
71024	10	0.3%
108012	5	0.2%
99856	5	0.2%
69149	5	0.2%
67106	5	0.2%
114211	5	0.2%
298537	5	0.2%
151760	5	0.2%
335423	5	0.2%
Other values (584)	2831	97.9%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
63215	5	0.2%
63651	5	0.2%
63792	5	0.2%
64609	5	0.2%
64819	4	0.1%

Value	Count	Frequency (%)
8550405	5	0.2%
3971896	5	0.2%
2720556	5	0.2%
2298628	5	0.2%
1567442	5	0.2%

Number of Veterans
Real number (ℝ_≥0)

Distinct count	577
Unique (%)	20.0%
Missing	13
Missing (%)	0.4%
Infinite	0
Infinite (%)	0.0%

Mean	9367.832522585128
Minimum	416.0
Maximum	156961.0
Zeros	0
Zeros (%)	0.0%
Memory size	22.6 KiB

Quantile statistics

Minimum	416
5-th percentile	1990
Q1	3739
median	5397
Q3	9368
95-th percentile	29511
Maximum	156961
Range	156545
Interquartile range (IQR)	5629

Descriptive statistics

Standard deviation	13211.21992
Coefficient of variation (CV)	1.410274991
Kurtosis	39.85381815
Mean	9367.832523
Median Absolute Deviation (MAD)	2281
Skewness	5.295923255
Sum	26960622
Variance	174536331.9

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
4211	10	0.3%
3397	10	0.3%
3647	10	0.3%
3116	10	0.3%
3063	10	0.3%
5532	10	0.3%
5714	10	0.3%
5204	10	0.3%
3027	9	0.3%
3404	9	0.3%
Other values (567)	2780	96.2%
(Missing)	13	0.4%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
416	3	0.1%
629	5	0.2%
693	4	0.1%
705	5	0.2%
724	5	0.2%

Value	Count	Frequency (%)
156961	5	0.2%
109089	5	0.2%
92489	5	0.2%
85417	5	0.2%
75432	5	0.2%

Foreign-born
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct count	587
Unique (%)	20.4%
Missing	13
Missing (%)	0.4%
Infinite	0
Infinite (%)	0.0%

Mean	40653.598679638635
Minimum	861.0
Maximum	3212500.0
Zeros	0
Zeros (%)	0.0%
Memory size	22.6 KiB

Quantile statistics

Minimum	861
5-th percentile	3215
Q1	9224
median	18822
Q3	33971.75
95-th percentile	109222.15
Maximum	3212500
Range	3211639
Interquartile range (IQR)	24747.75

Descriptive statistics

Standard deviation	155749.1037
Coefficient of variation (CV)	3.831127101
Kurtosis	310.3878352
Mean	40653.59868
Median Absolute Deviation (MAD)	11101
Skewness	16.35579487
Sum	117001057
Variance	2.425778329e+10

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
5757	10	0.3%
13409	10	0.3%
9735	5	0.2%
13253	5	0.2%
22958	5	0.2%
6275	5	0.2%
18451	5	0.2%
10516	5	0.2%
6123	5	0.2%
33258	5	0.2%
Other values (577)	2818	97.5%
(Missing)	13	0.4%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
861	5	0.2%
1058	5	0.2%
1062	4	0.1%
1224	5	0.2%
1531	5	0.2%

Value	Count	Frequency (%)
3212500	5	0.2%
1485425	5	0.2%
696210	5	0.2%
573463	5	0.2%
401493	5	0.2%

Average Household Size
Real number (ℝ_≥0)

Distinct count	161
Unique (%)	5.6%
Missing	16
Missing (%)	0.6%
Infinite	0
Infinite (%)	0.0%

Mean	2.742542608695652
Minimum	2.0
Maximum	4.98
Zeros	0
Zeros (%)	0.0%
Memory size	22.6 KiB

Quantile statistics

Minimum	2
5-th percentile	2.22
Q1	2.43
median	2.65
Q3	2.95
95-th percentile	3.58
Maximum	4.98
Range	2.98
Interquartile range (IQR)	0.52

Descriptive statistics

Standard deviation	0.4332910879
Coefficient of variation (CV)	0.1579888263
Kurtosis	2.861908247
Mean	2.742542609
Median Absolute Deviation (MAD)	0.24
Skewness	1.409563879
Sum	7884.81
Variance	0.1877411669

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
2.4	78	2.7%
2.72	68	2.4%
2.97	55	1.9%
2.64	54	1.9%
2.39	54	1.9%
2.41	50	1.7%
2.68	50	1.7%
2.52	49	1.7%
2.73	48	1.7%
2.55	45	1.6%
Other values (151)	2324	80.4%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
2	5	0.2%
2.06	5	0.2%
2.08	10	0.3%
2.1	4	0.1%
2.11	5	0.2%

Value	Count	Frequency (%)
4.98	5	0.2%
4.78	4	0.1%
4.58	5	0.2%
4.57	3	0.1%
4.43	4	0.1%

State Code
Categorical

HIGH CORRELATION

Distinct count	49
Unique (%)	1.7%
Missing	0
Missing (%)	0.0%
Memory size	22.6 KiB

CA	676
TX	273
FL	222
IL	91
WA	85
Other values (44)	1544

Common Values
Length

Value	Count	Frequency (%)
CA	676	23.4%
TX	273	9.4%
FL	222	7.7%
IL	91	3.1%
WA	85	2.9%
AZ	80	2.8%
CO	80	2.8%
MI	79	2.7%
VA	70	2.4%
NC	70	2.4%
Other values (39)	1165	40.3%

Length

Max length	2
Median length	2
Mean length	2
Min length	2

Race
Categorical

Distinct count	5
Unique (%)	0.2%
Missing	0
Missing (%)	0.0%
Memory size	22.6 KiB

Hispanic or Latino	596
White	589
Black or African-American	584
Asian	583
American Indian and Alaska Native	539

Common Values
Length

Value	Count	Frequency (%)
Hispanic or Latino	596	20.6%
White	589	20.4%
Black or African-American	584	20.2%
Asian	583	20.2%
American Indian and Alaska Native	539	18.6%

Length

Max length	33
Median length	18
Mean length	16.94050502
Min length	5

Count
Real number (ℝ_≥0)

Distinct count	2785
Unique (%)	96.3%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	48963.77447250087
Minimum	98
Maximum	3835726
Zeros	0
Zeros (%)	0.0%
Memory size	22.6 KiB

Quantile statistics

Minimum	98
5-th percentile	778.5
Q1	3435
median	13780
Q3	54447
95-th percentile	162670.5
Maximum	3835726
Range	3835628
Interquartile range (IQR)	51012

Descriptive statistics

Standard deviation	144385.5886
Coefficient of variation (CV)	2.94882472
Kurtosis	246.5728171
Mean	48963.77447
Median Absolute Deviation (MAD)	12231
Skewness	12.97352625
Sum	141554272
Variance	2.084719819e+10

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
881	3	0.1%
1343	3	0.1%
6547	3	0.1%
1615	3	0.1%
251	3	0.1%
535	3	0.1%
876	3	0.1%
713	3	0.1%
2452	2	0.1%
11815	2	0.1%
Other values (2775)	2863	99.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
98	1	< 0.1%
128	1	< 0.1%
158	1	< 0.1%
182	1	< 0.1%
203	1	< 0.1%

Value	Count	Frequency (%)
3835726	1	< 0.1%
2485125	1	< 0.1%
2192248	1	< 0.1%
2177650	1	< 0.1%
1936732	1	< 0.1%

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

Phik (φk)

Phik (φk) is a new and practical correlation coefficient that works consistently between categorical, ordinal and interval variables, captures non-linear dependency and reverts to the Pearson correlation coefficient in case of a bivariate normal input distribution. There is extensive documentation available here.

Cramér's V (φc)

Cramér's V is an association measure for nominal random variables. The coefficient ranges from 0 to 1, with 0 indicating independence and 1 indicating perfect association. The empirical estimators used for Cramér's V have been proved to be biased, even for large samples. We use a bias-corrected measure that has been proposed by Bergsma in 2013 that can be found here.

First rows

	City	State	Median Age	Male Population	Female Population	Total Population	Number of Veterans	Foreign-born	Average Household Size	State Code	Race	Count
0	Silver Spring	Maryland	33.8	40601.0	41862.0	82463	1562.0	30908.0	2.60	MD	Hispanic or Latino	25924
1	Quincy	Massachusetts	41.0	44129.0	49500.0	93629	4147.0	32935.0	2.39	MA	White	58723
2	Hoover	Alabama	38.5	38040.0	46799.0	84839	4819.0	8229.0	2.58	AL	Asian	4759
3	Rancho Cucamonga	California	34.5	88127.0	87105.0	175232	5821.0	33878.0	3.18	CA	Black or African-American	24437
4	Newark	New Jersey	34.6	138040.0	143873.0	281913	5829.0	86253.0	2.73	NJ	White	76402
5	Peoria	Illinois	33.1	56229.0	62432.0	118661	6634.0	7517.0	2.40	IL	American Indian and Alaska Native	1343
6	Avondale	Arizona	29.1	38712.0	41971.0	80683	4815.0	8355.0	3.18	AZ	Black or African-American	11592
7	West Covina	California	39.8	51629.0	56860.0	108489	3800.0	37038.0	3.56	CA	Asian	32716
8	O'Fallon	Missouri	36.0	41762.0	43270.0	85032	5783.0	3269.0	2.77	MO	Hispanic or Latino	2583
9	High Point	North Carolina	35.5	51751.0	58077.0	109828	5204.0	16315.0	2.65	NC	Asian	11060

Last rows

	City	State	Median Age	Male Population	Female Population	Total Population	Number of Veterans	Foreign-born	Average Household Size	State Code	Race	Count
2881	Gulfport	Mississippi	35.1	33108.0	38764.0	71872	6646.0	3072.0	2.54	MS	White	42870
2882	Davis	California	26.3	33493.0	34163.0	67656	2176.0	13997.0	2.69	CA	American Indian and Alaska Native	779
2883	Los Angeles	California	35.0	1958998.0	2012898.0	3971896	85417.0	1485425.0	2.86	CA	Black or African-American	404868
2884	Mount Vernon	New York	38.5	31876.0	36745.0	68621	2064.0	23777.0	2.85	NY	Hispanic or Latino	9446
2885	Lynchburg	Virginia	28.7	38614.0	41198.0	79812	4322.0	4364.0	2.48	VA	White	53727
2886	Stockton	California	32.5	150976.0	154674.0	305650	12822.0	79583.0	3.16	CA	American Indian and Alaska Native	19834
2887	Southfield	Michigan	41.6	31369.0	41808.0	73177	4035.0	4011.0	2.27	MI	American Indian and Alaska Native	983
2888	Indianapolis	Indiana	34.1	410615.0	437808.0	848423	42186.0	72456.0	2.53	IN	White	553665
2889	Somerville	Massachusetts	31.0	41028.0	39306.0	80334	2103.0	22292.0	2.43	MA	American Indian and Alaska Native	374
2890	Coral Springs	Florida	37.2	63316.0	66186.0	129502	4724.0	38552.0	3.17	FL	White	90896

Overview

Variables

Interactions

Correlations

Pearson's r

Spearman's ρ

Kendall's τ

Phik (φk)

Cramér's V (φc)

Missing values

Sample

First rows

Last rows